WebXR Hit Test Result: Ray Casting Result Processing for Immersive Experiences

The WebXR Device API opens up exciting possibilities for creating immersive augmented reality (AR) and virtual reality (VR) experiences directly within the browser. One of the fundamental aspects of building interactive WebXR applications is understanding and effectively utilizing hit test results. This blog post provides a comprehensive guide to processing hit test results obtained through ray casting, enabling you to create intuitive and engaging user interactions within your WebXR scenes.

What is Ray Casting and Why is it Important in WebXR?

Ray casting is a technique used to determine if a ray, originating from a specific point and direction, intersects with objects in a 3D scene. In WebXR, ray casting is typically used to simulate the user's gaze or the trajectory of a virtual object. When the ray intersects with a real-world surface (in AR) or a virtual object (in VR), a hit test result is generated.

Hit test results are crucial for several reasons:

• Placement of Virtual Objects: In AR, hit tests allow you to accurately place virtual objects onto real-world surfaces, such as tables, floors, or walls.
• User Interaction: By tracking where the user is looking or pointing, hit tests enable interactions with virtual objects, such as selecting, manipulating, or activating them.
• Navigation: In VR environments, hit tests can be used to implement navigation systems, allowing users to teleport or move around the scene by pointing at specific locations.
• Collision Detection: Hit tests can be used for basic collision detection, determining when a virtual object collides with another object or the real world.

Understanding the WebXR Hit Test API

The WebXR Hit Test API provides the necessary tools for performing ray casting and obtaining hit test results. Here's a breakdown of the key concepts and functions:

XRRay

An XRRay represents a ray in 3D space. It is defined by an origin point and a direction vector. You can create an XRRay using the XRFrame.getPose() method, which returns the pose of a tracked input source (e.g., the user's head, a hand controller). From the pose, you can derive the ray's origin and direction.

XRHitTestSource

An XRHitTestSource represents a source of hit test results. You create a hit test source using the XRSession.requestHitTestSource() or XRSession.requestHitTestSourceForTransientInput() method. The first method is generally used for continuous hit testing based on a persistent source, such as the user's head position, whereas the second is intended for transient input events, like button presses or gestures.

XRHitTestResult

An XRHitTestResult represents a single intersection point between the ray and a surface. It contains information about the intersection, such as the distance from the ray origin to the hit point and the pose of the hit point in the scene's reference space.

XRHitTestResult.getPose()

This method returns the XRPose of the hit point. The pose contains the position and orientation of the hit point, which can be used to place virtual objects or perform other transformations.

Processing Hit Test Results: A Step-by-Step Guide

Let's walk through the process of obtaining and processing hit test results in a WebXR application. This example assumes you are using a rendering library like three.js or Babylon.js.

1. Requesting a Hit Test Source

First, you need to request a hit test source from the XRSession. This is typically done after the session has started. You'll need to specify the coordinate system in which you want the hit test results to be returned. For example:

            let xrHitTestSource = null;

async function createHitTestSource(xrSession) {
  try {
    xrHitTestSource = await xrSession.requestHitTestSource({
      space: xrSession.viewerSpace // Or xrSession.local
    });
  } catch (error) {
    console.error("Failed to create hit test source: ", error);
  }
}

// Call this function after the XR session has started
// createHitTestSource(xrSession);

            

              
                Copy
              
            
          

Explanation:

• xrSession.requestHitTestSource(): This function requests a hit test source from the XR session.
• { space: xrSession.viewerSpace }: This specifies the coordinate system in which the hit test results will be returned. viewerSpace is relative to the viewer's position, while local is relative to the XR origin. You can also use localFloor for tracking relative to the floor.
• Error handling: The try...catch block ensures that errors during the hit test source creation are caught and logged.

2. Performing the Hit Test in the Animation Loop

Inside your animation loop (the function that renders each frame), you'll need to perform the hit test using the XRFrame.getHitTestResults() method. This method returns an array of XRHitTestResult objects, representing all the intersections found in the scene.

            function onXRFrame(time, frame) {
  const session = frame.session;
  session.requestAnimationFrame(onXRFrame);

  const pose = frame.getViewerPose(xrSession.referenceSpace);

  if (pose) {
    if (xrHitTestSource) {
      const hitTestResults = frame.getHitTestResults(xrHitTestSource);

      if (hitTestResults.length > 0) {
        processHitTestResults(hitTestResults);
      }
    }
  }

  renderer.render(scene, camera);
}

            

              
                Copy
              
            
          

Explanation:

• frame.getViewerPose(xrSession.referenceSpace): Gets the pose of the viewer (headset). This is necessary to know where the viewer is and where they are looking.
• frame.getHitTestResults(xrHitTestSource): Performs the hit test using the previously created hit test source.
• hitTestResults.length > 0: Checks if any intersections were found.

3. Processing the Hit Test Results

The processHitTestResults() function is where you'll handle the results of the hit test. This typically involves updating the position and orientation of a virtual object based on the hit point's pose.

            function processHitTestResults(hitTestResults) {
  const hit = hitTestResults[0]; // Get the first hit result
  const hitPose = hit.getPose(xrSession.referenceSpace);

  if (hitPose) {
    // Update the position and orientation of a virtual object
    virtualObject.position.set(hitPose.transform.position.x, hitPose.transform.position.y, hitPose.transform.position.z);
    virtualObject.quaternion.set(hitPose.transform.orientation.x, hitPose.transform.orientation.y, hitPose.transform.orientation.z, hitPose.transform.orientation.w);

    // Show visual feedback (e.g., a circle) at the hit point
    hitMarker.position.set(hitPose.transform.position.x, hitPose.transform.position.y, hitPose.transform.position.z);
    hitMarker.quaternion.set(hitPose.transform.orientation.x, hitPose.transform.orientation.y, hitPose.transform.orientation.z, hitPose.transform.orientation.w);
    hitMarker.visible = true;
  } else {
    hitMarker.visible = false;
  }
}

            

              
                Copy
              
            
          

Explanation:

• hitTestResults[0]: Retrieves the first hit test result. If multiple intersections are possible, you might need to iterate through the entire array and choose the most appropriate result based on your application's logic.
• hit.getPose(xrSession.referenceSpace): Gets the pose of the hit point in the specified reference space.
• virtualObject.position.set(...) and virtualObject.quaternion.set(...): Update the position and rotation (quaternion) of a virtual object (e.g., a three.js Mesh) to match the hit point's pose.
• Visual Feedback: The example also includes code to show visual feedback at the hit point, such as a circle or a simple marker, to help the user understand where they are interacting with the scene.

Advanced Hit Testing Techniques

Beyond the basic example above, there are several advanced techniques you can use to enhance your hit testing implementations:

Hit Testing with Transient Input

For interactions triggered by transient input, such as button presses or hand gestures, you can use the XRSession.requestHitTestSourceForTransientInput() method. This method creates a hit test source that is specific to a single input event. This is useful for avoiding unintended interactions based on continuous hit testing.

            async function handleSelect(event) {
  try {
    const frame = event.frame;
    const inputSource = event.inputSource;

    const hitTestResults = await frame.getHitTestResultsForTransientInput(inputSource, {
      profile: 'generic-touchscreen', // Or the appropriate input profile
      space: xrSession.viewerSpace
    });

    if (hitTestResults.length > 0) {
      processHitTestResults(hitTestResults);
    }
  } catch (error) {
    console.error("Error during transient hit test: ", error);
  }
}

// Attach this function to your input select event listener
// xrSession.addEventListener('select', handleSelect);

            

              
                Copy
              
            
          

Filtering Hit Test Results

In some cases, you might want to filter hit test results based on specific criteria, such as the distance from the ray origin or the type of surface that was intersected. You can achieve this by manually filtering the XRHitTestResult array after obtaining it.

            function processHitTestResults(hitTestResults) {
  const filteredResults = hitTestResults.filter(result => {
    const hitPose = result.getPose(xrSession.referenceSpace);
    if (!hitPose) return false; // Skip if no pose

    const distance = Math.sqrt(
      Math.pow(hitPose.transform.position.x - camera.position.x, 2) +
      Math.pow(hitPose.transform.position.y - camera.position.y, 2) +
      Math.pow(hitPose.transform.position.z - camera.position.z, 2)
    );

    return distance < 2; // Only consider hits within 2 meters
  });

  if (filteredResults.length > 0) {
    const hit = filteredResults[0];
    const hitPose = hit.getPose(xrSession.referenceSpace);

    if (hitPose) {
      // Update object position based on the filtered result
      virtualObject.position.set(hitPose.transform.position.x, hitPose.transform.position.y, hitPose.transform.position.z);
      virtualObject.quaternion.set(hitPose.transform.orientation.x, hitPose.transform.orientation.y, hitPose.transform.orientation.z, hitPose.transform.orientation.w);
    }
  }
}

            

              
                Copy
              
            
          

Using Different Reference Spaces

The choice of reference space (viewerSpace, local, localFloor, or other custom spaces) significantly impacts how hit test results are interpreted. Consider the following:

• viewerSpace: Provides results relative to the viewer's position. This is useful for creating interactions that are directly tied to the user's gaze.
• local: Provides results relative to the XR origin (the starting point of the XR session). This is suitable for creating experiences where objects remain fixed in the physical environment.
• localFloor: Similar to local, but the Y-axis is aligned with the floor. This simplifies the process of placing objects on the floor.

Choose the reference space that best aligns with your application's requirements. Experiment with different reference spaces to understand their behavior and limitations.

Optimization Strategies for Hit Testing

Hit testing can be a computationally intensive process, especially in complex scenes. Here are some optimization strategies to consider:

• Limit the Frequency of Hit Tests: Perform hit tests only when necessary, rather than every frame. For example, you could perform hit tests only when the user is actively interacting with the scene.
• Use a Bounding Volume Hierarchy (BVH): If you're performing hit tests against a large number of objects, consider using a BVH to accelerate the intersection calculations. Libraries like three.js and Babylon.js provide built-in BVH implementations.
• Spatial Partitioning: Divide the scene into smaller regions and perform hit tests only against the regions that are likely to contain intersections. This can significantly reduce the number of objects that need to be checked.
• Reduce Polygon Count: Simplify the geometry of your models to reduce the number of polygons that need to be tested. This can improve performance, especially on mobile devices.
• WebWorker: offload the computation to a web worker to ensure the hit test process doesnt lock the main thread.

Cross-Platform Considerations

WebXR aims to be cross-platform, but there may be subtle differences in behavior across different devices and browsers. Keep the following in mind:

• Device Capabilities: Not all devices support all WebXR features. Use feature detection to determine which features are available and adapt your application accordingly.
• Input Profiles: Different devices may use different input profiles (e.g., generic-touchscreen, hand-tracking, gamepad). Ensure that your application supports multiple input profiles and provides appropriate fallback mechanisms.
• Performance: Performance can vary significantly across different devices. Optimize your application for the lowest-end devices you plan to support.
• Browser Compatibility: Ensure your app is tested and works across major browsers like Chrome, Firefox, and Edge.

Global Examples of WebXR Applications Using Hit Testing

Here are some examples of WebXR applications that effectively utilize hit testing to create compelling and intuitive user experiences:

• IKEA Place (Sweden): Allows users to virtually place IKEA furniture in their homes using AR. Hit testing is used to accurately position the furniture on the floor and other surfaces.
• Sketchfab AR (France): Enables users to view 3D models from Sketchfab in AR. Hit testing is used to place the models in the real world.
• Augmented Images (Various): Many AR applications use image tracking combined with hit testing to anchor virtual content to specific images or markers in the real world.
• WebXR Games (Global): Numerous games are being developed using WebXR, many of which rely on hit testing for object placement, interaction, and navigation.
• Virtual Tours (Global): Immersive tours of locations, museums, or properties often employ hit testing for user navigation and interactive elements within the virtual environment.

Conclusion

Mastering WebXR hit test results and ray casting processing is essential for creating compelling and intuitive AR and VR experiences on the web. By understanding the underlying concepts and applying the techniques described in this blog post, you can build immersive applications that seamlessly blend the virtual and real worlds, or create engaging virtual environments with natural and intuitive user interactions. Remember to optimize your hit testing implementation for performance and consider cross-platform compatibility to ensure a smooth experience for all users. As the WebXR ecosystem continues to evolve, expect further advancements and refinements to the hit testing API, opening up even more creative possibilities for immersive web development. Always consult the latest WebXR specifications and browser documentation for the most up-to-date information.